{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter Optimization (HPO) of Machine Learning Models\n",
    "L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.\n",
    "\n",
    "### **Sample code for classification problems**  \n",
    "**Dataset used:**  \n",
    "&nbsp; MNIST from sklearn\n",
    "\n",
    "**Machine learning algorithms used:**  \n",
    "&nbsp; Random forest (RF), support vector machine (SVM), k-nearest neighbor (KNN), artificial neural network (ANN)\n",
    "\n",
    "**HPO algorithms used:**  \n",
    "&nbsp; Grid search, random search, hyperband, Bayesian Optimization with Gaussian Processes (BO-GP), Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE), particle swarm optimization (PSO), genetic algorithm (GA).\n",
    "\n",
    "**Performance metric:**  \n",
    "&nbsp; Classification accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "from sklearn.ensemble import RandomForestClassifier,RandomForestRegressor\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier,KNeighborsRegressor\n",
    "from sklearn.svm import SVC,SVR\n",
    "from sklearn import datasets\n",
    "import scipy.stats as stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load MNIST dataset\n",
    "The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The MNIST database has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.\n",
    "\n",
    "You can read more about the data and the variables [[1]](http://yann.lecun.com/exdb/mnist/) [[2]](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d = datasets.load_digits()\n",
    "X = d.data\n",
    "y = d.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': array([[ 0.,  0.,  5., ...,  0.,  0.,  0.],\n",
       "        [ 0.,  0.,  0., ..., 10.,  0.,  0.],\n",
       "        [ 0.,  0.,  0., ..., 16.,  9.,  0.],\n",
       "        ...,\n",
       "        [ 0.,  0.,  1., ...,  6.,  0.,  0.],\n",
       "        [ 0.,  0.,  2., ..., 12.,  0.,  0.],\n",
       "        [ 0.,  0., 10., ..., 12.,  1.,  0.]]),\n",
       " 'target': array([0, 1, 2, ..., 8, 9, 8]),\n",
       " 'frame': None,\n",
       " 'feature_names': ['pixel_0_0',\n",
       "  'pixel_0_1',\n",
       "  'pixel_0_2',\n",
       "  'pixel_0_3',\n",
       "  'pixel_0_4',\n",
       "  'pixel_0_5',\n",
       "  'pixel_0_6',\n",
       "  'pixel_0_7',\n",
       "  'pixel_1_0',\n",
       "  'pixel_1_1',\n",
       "  'pixel_1_2',\n",
       "  'pixel_1_3',\n",
       "  'pixel_1_4',\n",
       "  'pixel_1_5',\n",
       "  'pixel_1_6',\n",
       "  'pixel_1_7',\n",
       "  'pixel_2_0',\n",
       "  'pixel_2_1',\n",
       "  'pixel_2_2',\n",
       "  'pixel_2_3',\n",
       "  'pixel_2_4',\n",
       "  'pixel_2_5',\n",
       "  'pixel_2_6',\n",
       "  'pixel_2_7',\n",
       "  'pixel_3_0',\n",
       "  'pixel_3_1',\n",
       "  'pixel_3_2',\n",
       "  'pixel_3_3',\n",
       "  'pixel_3_4',\n",
       "  'pixel_3_5',\n",
       "  'pixel_3_6',\n",
       "  'pixel_3_7',\n",
       "  'pixel_4_0',\n",
       "  'pixel_4_1',\n",
       "  'pixel_4_2',\n",
       "  'pixel_4_3',\n",
       "  'pixel_4_4',\n",
       "  'pixel_4_5',\n",
       "  'pixel_4_6',\n",
       "  'pixel_4_7',\n",
       "  'pixel_5_0',\n",
       "  'pixel_5_1',\n",
       "  'pixel_5_2',\n",
       "  'pixel_5_3',\n",
       "  'pixel_5_4',\n",
       "  'pixel_5_5',\n",
       "  'pixel_5_6',\n",
       "  'pixel_5_7',\n",
       "  'pixel_6_0',\n",
       "  'pixel_6_1',\n",
       "  'pixel_6_2',\n",
       "  'pixel_6_3',\n",
       "  'pixel_6_4',\n",
       "  'pixel_6_5',\n",
       "  'pixel_6_6',\n",
       "  'pixel_6_7',\n",
       "  'pixel_7_0',\n",
       "  'pixel_7_1',\n",
       "  'pixel_7_2',\n",
       "  'pixel_7_3',\n",
       "  'pixel_7_4',\n",
       "  'pixel_7_5',\n",
       "  'pixel_7_6',\n",
       "  'pixel_7_7'],\n",
       " 'target_names': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
       " 'images': array([[[ 0.,  0.,  5., ...,  1.,  0.,  0.],\n",
       "         [ 0.,  0., 13., ..., 15.,  5.,  0.],\n",
       "         [ 0.,  3., 15., ..., 11.,  8.,  0.],\n",
       "         ...,\n",
       "         [ 0.,  4., 11., ..., 12.,  7.,  0.],\n",
       "         [ 0.,  2., 14., ..., 12.,  0.,  0.],\n",
       "         [ 0.,  0.,  6., ...,  0.,  0.,  0.]],\n",
       " \n",
       "        [[ 0.,  0.,  0., ...,  5.,  0.,  0.],\n",
       "         [ 0.,  0.,  0., ...,  9.,  0.,  0.],\n",
       "         [ 0.,  0.,  3., ...,  6.,  0.,  0.],\n",
       "         ...,\n",
       "         [ 0.,  0.,  1., ...,  6.,  0.,  0.],\n",
       "         [ 0.,  0.,  1., ...,  6.,  0.,  0.],\n",
       "         [ 0.,  0.,  0., ..., 10.,  0.,  0.]],\n",
       " \n",
       "        [[ 0.,  0.,  0., ..., 12.,  0.,  0.],\n",
       "         [ 0.,  0.,  3., ..., 14.,  0.,  0.],\n",
       "         [ 0.,  0.,  8., ..., 16.,  0.,  0.],\n",
       "         ...,\n",
       "         [ 0.,  9., 16., ...,  0.,  0.,  0.],\n",
       "         [ 0.,  3., 13., ..., 11.,  5.,  0.],\n",
       "         [ 0.,  0.,  0., ..., 16.,  9.,  0.]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[ 0.,  0.,  1., ...,  1.,  0.,  0.],\n",
       "         [ 0.,  0., 13., ...,  2.,  1.,  0.],\n",
       "         [ 0.,  0., 16., ..., 16.,  5.,  0.],\n",
       "         ...,\n",
       "         [ 0.,  0., 16., ..., 15.,  0.,  0.],\n",
       "         [ 0.,  0., 15., ..., 16.,  0.,  0.],\n",
       "         [ 0.,  0.,  2., ...,  6.,  0.,  0.]],\n",
       " \n",
       "        [[ 0.,  0.,  2., ...,  0.,  0.,  0.],\n",
       "         [ 0.,  0., 14., ..., 15.,  1.,  0.],\n",
       "         [ 0.,  4., 16., ..., 16.,  7.,  0.],\n",
       "         ...,\n",
       "         [ 0.,  0.,  0., ..., 16.,  2.,  0.],\n",
       "         [ 0.,  0.,  4., ..., 16.,  2.,  0.],\n",
       "         [ 0.,  0.,  5., ..., 12.,  0.,  0.]],\n",
       " \n",
       "        [[ 0.,  0., 10., ...,  1.,  0.,  0.],\n",
       "         [ 0.,  2., 16., ...,  1.,  0.,  0.],\n",
       "         [ 0.,  0., 15., ..., 15.,  0.,  0.],\n",
       "         ...,\n",
       "         [ 0.,  4., 16., ..., 16.,  6.,  0.],\n",
       "         [ 0.,  8., 16., ..., 16.,  8.,  0.],\n",
       "         [ 0.,  1.,  8., ..., 12.,  1.,  0.]]]),\n",
       " 'DESCR': \".. _digits_dataset:\\n\\nOptical recognition of handwritten digits dataset\\n--------------------------------------------------\\n\\n**Data Set Characteristics:**\\n\\n    :Number of Instances: 1797\\n    :Number of Attributes: 64\\n    :Attribute Information: 8x8 image of integer pixels in the range 0..16.\\n    :Missing Attribute Values: None\\n    :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)\\n    :Date: July; 1998\\n\\nThis is a copy of the test set of the UCI ML hand-written digits datasets\\nhttps://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits\\n\\nThe data set contains images of hand-written digits: 10 classes where\\neach class refers to a digit.\\n\\nPreprocessing programs made available by NIST were used to extract\\nnormalized bitmaps of handwritten digits from a preprinted form. From a\\ntotal of 43 people, 30 contributed to the training set and different 13\\nto the test set. 32x32 bitmaps are divided into nonoverlapping blocks of\\n4x4 and the number of on pixels are counted in each block. This generates\\nan input matrix of 8x8 where each element is an integer in the range\\n0..16. This reduces dimensionality and gives invariance to small\\ndistortions.\\n\\nFor info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.\\nT. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.\\nL. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,\\n1994.\\n\\n.. topic:: References\\n\\n  - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their\\n    Applications to Handwritten Digit Recognition, MSc Thesis, Institute of\\n    Graduate Studies in Science and Engineering, Bogazici University.\\n  - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.\\n  - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.\\n    Linear dimensionalityreduction using relevance weighted LDA. School of\\n    Electrical and Electronic Engineering Nanyang Technological University.\\n    2005.\\n  - Claudio Gentile. A New Approximate Maximal Margin Classification\\n    Algorithm. NIPS. 2000.\\n\"}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAACXCAYAAAARS4GeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAALBUlEQVR4nO3dX2yd510H8O+vi8ooW2tnE0wU1sSdBAK0mqZTmZBQqjnSuJgcMRJNDDRXmhJxA5G4cG5gjsZQghByxYYWEGoZMFgjIJ2QCmq0uqMXgGLhTipsF2lamNikQp1uHfsjwcvFcUbUpmnzvufkxE8+HymSz+n5vs9j95dzvnlfH7u6rgsAQMtumvYGAAAmTeEBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeU0Xnqp6tKo+NO7HcmMxRwxlhhgHczRMXW8/h6eqXrrk5i1Jvp3kf7ZuH+667s+u/a7Gq6rek+QTSd6e5B+TLHVd99x0d9WW1ueoqm5O8ukk9yS5I8l9XdetTXVTjbkBZuinknw0yZ6MPq+1JL/Sdd1Xprmv1twAc/RjST6V5M6tu9YzmqN/md6uLu+6O8PTdd2bLv5J8m9J3nfJfd8djKraMb1d9ldVb03yV0l+PcnOJGeTfGaqm2pQ63O05ckkv5jkq9PeSItugBmaTfIHSXZlVJq/nuTBaW6oRTfAHP1Hkp/P6PXsrUk+m+QvprqjV3HdFZ5XU1V7q+rLVbVcVV9N8mBVzVbV31TV81W1ufXxD12SWauqD299vFRVT1bV72w99nxV/WzPx+6uqs9X1der6kxVfaKq/vR1fio/l+TprutOdV33rSQrSe6qqh8d/lXitbQyR13XfafrutWu657M//9rkWugoRl6dOt56Gtd1/13ko8n+ekxfZl4DQ3N0YWu657tRpeLKqPno3eM56s0Xtum8Gx5W0Yt8o4khzLa/4Nbt9+e5JsZ/aV9Nfcm+VJGLfS3k/xRVVWPx346yT8leUtGheWXLg1W1Req6hde5bg/nuSpize6rvtGknNb93NttDBHTFeLM/QzSZ5+nY9lPJqZo6q6kORbSX4vyW9d6bHTst1Oof1vko90XfftrdvfTPKXF/9jVX0syeNXyD/Xdd0fbj32j5P8fpIfyOUvCVz2sTX63ol3JXlP13XfSfJkVX320mDXde+8wh7elOT5l933YpI3XyHDeLUwR0xXUzNUVe9M8htJFl/P4xmbZuao67qZqvq+JB9Kcl1+T+p2O8Pz/NZloCRJVd1SVSer6rmq+lqSzyeZqao3vEr+u0OwdQo3GRWQq3nsDyZ54ZL7kuTfr+JzeCnJrS+779aMrp9zbbQwR0xXMzNUVe9I8miSX+267u+vNs8gzczR1nG/keSTST5VVd/f5xiTtN0Kz8vfUvZrSX4kyb1d192a0SnZZHQdcVK+kmRnVd1yyX0/fBX5p5PcdfHGViO+M04lX0stzBHT1cQMVdUdSc4k+WjXdX8yzs3xujQxRy9zU0bvRrt90K4mYLsVnpd7c0anAC9U1c4kH5n0gltvHz+bZKWqbq6qdyd531Uc4q+T/ERVvb+q3pjRaeQvdF33xQlsl9dnO85Rqup7tmYoSW6uqjde4fo9k7XtZqiqbk/yuSQf77rukxPaJldnO87Rvqr6yap6Q1XdmuR3k2wm+dfJ7Li/7V54VpN8b5L/TPIPSf72Gq37wSTvTvJfSX4zo7eVX7wGm6p6uqo+eLlg13XPJ3l/ko9lNBT3JvnApDfMFa1mm83Rli9l9OR4e5K/2/r4jontlitZzfaboQ8nmcvohe6li38mvWGuaDXbb45mkvx5Rt+Lei6jKxbvvfRS3fXiuvvBg9tRVX0myRe7rpt4G6dd5oihzBDj0OocbfczPFNRVe+qqjur6qaqem9G72w4PeVtsc2YI4YyQ4zDjTJH2+1t6deLt2X005LfkuTLSX6567p/nu6W2IbMEUOZIcbhhpgjl7QAgOa5pAUANO+1LmlN5fTPqVOnBuWXl5d7Z/ft29c7e/z48d7Z2dnZ3tkxmPRbmbflacS9e/f2zl64cKF39tixY72zi4tT/UG5k5yjbTlDa2trvbP79+/vnZ2fn++dHbLnMWjyuejEiROD8kePHu2d3b17d+/s+vp67+z1+JrmDA8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5Cg8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgObtmPYGLmd5eXlQ/vz5872zm5ubvbM7d+7snX344Yd7Z5PkwIEDg/K80szMTO/sE0880Tv7+OOP984uLi72zvJKGxsbg/L33Xdf7+xtt93WO/vss8/2znJ5R48e7Z0d+vx+8uTJ3tnDhw/3zq6vr/fOLiws9M5OijM8AEDzFB4AoHkKDwDQPIUHAGiewgMANE/hAQCap/AAAM1TeACA5ik8AEDzFB4AoHkKDwDQPIUHAGiewgMANE/hAQCat2NSBx7ya+XPnz8/aO1z5871zs7NzfXO7tu3r3d2yNcrSQ4cODAo36KNjY1B+bW1tbHs42rNz89PZV1e6fTp04Pyd911V+/s/v37e2ePHTvWO8vlHTp0qHd2eXl50Np79uzpnd29e3fv7MLCQu/s9cgZHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDm7ZjUgTc3N3tn77777kFrz83NDcr3tWfPnqms27LV1dXe2ZWVlUFrv/jii4Pyfe3du3cq6/JKR44cGZTftWvXVNZeXFzsneXyhryuPPPMM4PWPn/+fO/swsJC7+yQ1/HZ2dne2UlxhgcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPN2TOrAQ36t/L59+8a4k2tnyOc8Ozs7xp2048iRI72zS0tLg9ae1v+TCxcuTGXdVg35eq6urg5a+/Tp04PyfT300ENTWZfLm5ubG5R/4YUXemcXFhamkj1z5kzvbDKZ519neACA5ik8AEDzFB4AoHkKDwDQPIUHAGiewgMANE/hAQCap/AAAM1TeACA5ik8AEDzFB4AoHkKDwDQPIUHAGiewgMANG/HpA485Fe7r6+vj3EnV2dzc7N39uzZs72zBw8e7J2lLRsbG72z8/PzY9tHK1ZWVnpnH3jggfFt5CqdPn26d3ZmZmZs+2D6hryenjlzpnf28OHDvbMnTpzonU2S48ePD8pfjjM8AEDzFB4AoHkKDwDQPIUHAGiewgMANE/hAQCap/AAAM1TeACA5ik8AEDzFB4AoHkKDwDQPIUHAGiewgMANE/hAQCat2NSB56bm+udPXv27KC1T506NZXsEMvLy1NZF1q3tLTUO7u2tjZo7aeeeqp3dv/+/b2zi4uLvbP3339/7+zQtVt19OjRQfmFhYXe2c3Nzd7Zxx57rHf24MGDvbOT4gwPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPN2TOrAc3NzvbMnTpwYtPby8nLv7D333NM7u76+3jvL+M3MzAzKLy4u9s4+8sgjvbNra2u9s0tLS72zrZqfn++d3djYGLT2kPzKykrv7JD527VrV+9sMuzvTatmZ2cH5Q8dOjSmnVydgwcP9s6ePHlyjDsZD2d4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0r7qum/YeAAAmyhkeAKB5Cg8A0DyFBwBonsIDADRP4QEAmqfwAADN+z+hHt0iyNm/ygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x216 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, axes = plt.subplots(nrows=1, ncols=4, figsize=(10, 3))\n",
    "for ax, image, label in zip(axes, d.images, d.target):\n",
    "    ax.set_axis_off()\n",
    "    ax.imshow(image, cmap=plt.cm.gray_r, interpolation=\"nearest\")\n",
    "    ax.set_title(\"Training: %i\" % label)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Machine Learning Models: Classifiers with Default Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using 3-Fold Cross-Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.896553659132438\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "clf = RandomForestClassifier()\n",
    "clf.fit(X,y)\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='accuracy')\n",
    "print(\"Accuracy:\"+ str(scores.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.9705030401091262\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "clf = SVC()\n",
    "clf.fit(X,y)\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='accuracy')\n",
    "print(\"Accuracy:\"+ str(scores.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.9627317178438357\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "clf = KNeighborsClassifier()\n",
    "clf.fit(X,y)\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='accuracy')\n",
    "print(\"Accuracy:\"+ str(scores.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#ANN\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Input\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'sgd',neurons=32,batch_size=32,epochs=20,activation='relu',patience=3,loss='categorical_crossentropy'):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(neurons, input_shape=(X.shape[1],), activation=activation))\n",
    "    model.add(Dense(neurons, activation=activation))\n",
    "    model.add(Dense(10,activation='softmax'))  # 10 is the number of classes in the dataset, you can change it based on your dataset\n",
    "    model.compile(optimizer = optimizer, loss=loss)\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.9860879243183084\n"
     ]
    }
   ],
   "source": [
    "clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='accuracy')\n",
    "print(\"Accuracy:\"+ str(scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 1: Grid Search\n",
    "Search all the given hyper-parameter configurations\n",
    "\n",
    "**Advantages:**\n",
    "* Simple implementation.  \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Time-consuming,\n",
    "* Only efficient with categorical HPs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 30, 'max_depth': 15, 'criterion': 'entropy'}\n",
      "Accuracy:0.9332220367278798\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': [10, 20, 30],\n",
    "    #'max_features': ['sqrt',0.5],\n",
    "    'max_depth': [15,20,30,50],\n",
    "    #'min_samples_leaf': [1,2,4,8],\n",
    "    #\"bootstrap\":[True,False],\n",
    "    \"criterion\":['gini','entropy']\n",
    "}\n",
    "clf = RandomForestClassifier(random_state=0)\n",
    "grid = GridSearchCV(clf, rf_params, cv=3, scoring='accuracy')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"Accuracy:\"+ str(grid.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'rbf', 'C': 10}\n",
      "Accuracy:0.9744017807456873\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "rf_params = {\n",
    "    'C': [1,10, 100],\n",
    "    \"kernel\":['linear','poly','rbf','sigmoid']\n",
    "}\n",
    "clf = SVC(gamma='scale')\n",
    "grid = GridSearchCV(clf, rf_params, cv=3, scoring='accuracy')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"Accuracy:\"+ str(grid.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 3}\n",
      "Accuracy:0.9682804674457429\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "rf_params = {\n",
    "    'n_neighbors': [2, 3, 5,10,15,20],\n",
    "}\n",
    "clf = KNeighborsClassifier()\n",
    "grid = GridSearchCV(clf, rf_params, cv=3, scoring='accuracy')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"Accuracy:\"+ str(grid.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'relu', 'batch_size': 16, 'epochs': 50, 'neurons': 32, 'optimizer': 'adam', 'patience': 5}\n",
      "Accuracy:0.9994435169727324\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop','sgd'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'batch_size': [16,32],\n",
    "    'neurons':[16,32],\n",
    "    'epochs':[20,50],\n",
    "    'patience':[2,5]\n",
    "}\n",
    "clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "grid = GridSearchCV(clf, rf_params, cv=3,scoring='accuracy')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"Accuracy:\"+ str(grid.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 2: Random Search\n",
    "Randomly search hyper-parameter combinations in the search space\n",
    "\n",
    "**Advantages:**\n",
    "* More efficient than GS.\n",
    "* Enable parallelization. \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Not consider previous results.\n",
    "* Not efficient with conditional HPs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 36, 'max_features': 6, 'max_depth': 42, 'min_samples_split': 6, 'min_samples_leaf': 2, 'criterion': 'gini'}\n",
      "Accuracy:0.9309961046188091\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from scipy.stats import randint as sp_randint\n",
    "from random import randrange as sp_randrange\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': sp_randint(10,100),\n",
    "    \"max_features\":sp_randint(1,64),\n",
    "    'max_depth': sp_randint(5,50),\n",
    "    \"min_samples_split\":sp_randint(2,11),\n",
    "    \"min_samples_leaf\":sp_randint(1,11),\n",
    "    \"criterion\":['gini','entropy']\n",
    "}\n",
    "n_iter_search=20 #number of iterations is set to 20, you can increase this number if time permits\n",
    "clf = RandomForestClassifier(random_state=0)\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='accuracy')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"Accuracy:\"+ str(Random.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'rbf', 'C': 17.026713515892954}\n",
      "Accuracy:0.9744017807456873\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from scipy.stats import randint as sp_randint\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "rf_params = {\n",
    "    'C': stats.uniform(0,50),\n",
    "    \"kernel\":['linear','poly','rbf','sigmoid']\n",
    "}\n",
    "n_iter_search=20\n",
    "clf = SVC(gamma='scale')\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='accuracy')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"Accuracy:\"+ str(Random.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 4}\n",
      "Accuracy:0.9643850862548692\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from scipy.stats import randint as sp_randint\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "rf_params = {\n",
    "    'n_neighbors': range(1,20),\n",
    "}\n",
    "n_iter_search=10\n",
    "clf = KNeighborsClassifier()\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='accuracy')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"Accuracy:\"+ str(Random.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'relu', 'epochs': 20, 'optimizer': 'adam', 'patience': 8, 'batch_size': 16, 'neurons': 89}\n",
      "Accuracy:1.0\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from scipy.stats import randint as sp_randint\n",
    "from random import randrange as sp_randrange\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop','sgd'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':sp_randint(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':sp_randint(3,20)\n",
    "}\n",
    "n_iter_search=10\n",
    "clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='accuracy')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"Accuracy:\"+ str(Random.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 3: Hyperband\n",
    "Generate small-sized subsets and allocate budgets to each hyper-parameter combination based on its performance\n",
    "\n",
    "**Advantages:**\n",
    "* Enable parallelization.  \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Not efficient with conditional HPs.\n",
    "* Require subsets with small budgets to be representative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 100, 'max_features': 9, 'max_depth': 30, 'min_samples_split': 10, 'min_samples_leaf': 2, 'criterion': 'gini'}\n",
      "Accuracy:0.9321090706733445\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "from random import randrange as sp_randrange\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': sp_randint(10,100),\n",
    "    \"max_features\":sp_randint(1,64),\n",
    "    'max_depth': sp_randint(5,50),\n",
    "    \"min_samples_split\":sp_randint(2,11),\n",
    "    \"min_samples_leaf\":sp_randint(1,11),\n",
    "    \"criterion\":['gini','entropy']\n",
    "}\n",
    "clf = RandomForestClassifier(random_state=0)\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=10,max_iter=100,scoring='accuracy')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"Accuracy:\"+ str(hyper.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'rbf', 'C': 16}\n",
      "Accuracy:0.9744017807456873\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "from random import randrange as sp_randrange\n",
    "rf_params = {\n",
    "    'C': stats.uniform(0,50),\n",
    "    \"kernel\":['linear','poly','rbf','sigmoid']\n",
    "}\n",
    "clf = SVC(gamma='scale')\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=1,max_iter=50,scoring='accuracy',resource_param='C')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"Accuracy:\"+ str(hyper.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 2}\n",
      "Accuracy:0.9621591541457986\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "from random import randrange as sp_randrange\n",
    "rf_params = {\n",
    "    'n_neighbors': range(1,20),\n",
    "}\n",
    "clf = KNeighborsClassifier()\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=1,max_iter=20,scoring='accuracy',resource_param='n_neighbors')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"Accuracy:\"+ str(hyper.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'tanh', 'epochs': 10, 'batch_size': 16, 'patience': 7, 'optimizer': 'adam', 'neurons': 76}\n",
      "Accuracy:0.9994435169727324\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop','sgd'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':sp_randint(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':sp_randint(3,20)\n",
    "}\n",
    "clf = KerasClassifier(build_fn=ANN, epochs=20, verbose=0)\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=1,max_iter=10,scoring='accuracy',resource_param='epochs')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"Accuracy:\"+ str(hyper.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 4: BO-GP\n",
    "Bayesian Optimization with Gaussian Process (BO-GP)\n",
    "\n",
    "**Advantages:**\n",
    "* Fast convergence speed for continuous HPs.  \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Poor capacity for parallelization.\n",
    "* Not efficient with conditional HPs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using skopt.BayesSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 100, 'max_features': 1, 'max_depth': 19, 'min_samples_split': 2, 'min_samples_leaf': 1, 'criterion': 'gini'}\n",
      "Accuracy:0.9398998330550918\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': Integer(10,100),\n",
    "    \"max_features\":Integer(1,64),\n",
    "    'max_depth': Integer(5,50),\n",
    "    \"min_samples_split\":Integer(2,11),\n",
    "    \"min_samples_leaf\":Integer(1,11),\n",
    "    \"criterion\":['gini','entropy']\n",
    "}\n",
    "clf = RandomForestClassifier(random_state=0)\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=20, n_jobs=-1,scoring='accuracy')\n",
    "#number of iterations is set to 20, you can increase this number if time permits\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "bclf = Bayes.best_estimator_\n",
    "print(\"Accuracy:\"+ str(Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'rbf', 'C': 26.52140440440126}\n",
      "Accuracy:0.9744017807456873\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "rf_params = {\n",
    "    'C': Real(0.01,50),\n",
    "    \"kernel\":['linear','poly','rbf','sigmoid']\n",
    "}\n",
    "clf = SVC(gamma='scale')\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=20, n_jobs=-1,scoring='accuracy')\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "bclf = Bayes.best_estimator_\n",
    "print(\"Accuracy:\"+ str(Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 3}\n",
      "Accuracy:0.9682804674457429\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "rf_params = {\n",
    "    'n_neighbors': Integer(1,20),\n",
    "}\n",
    "clf = KNeighborsClassifier()\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=10, n_jobs=-1,scoring='accuracy')\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "bclf = Bayes.best_estimator_\n",
    "print(\"Accuracy:\"+ str(Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'tanh', 'epochs': 47, 'optimizer': 'adam', 'patience': 10, 'batch_size': 16, 'neurons': 54}\n",
      "Accuracy:1.0\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop','sgd'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':Integer(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':Integer(3,20)\n",
    "}\n",
    "clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=10, scoring='accuracy')\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "print(\"Accuracy:\"+ str(Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using skopt.gp_minimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.9293\n",
      "[100, 50, 1, 11, 1, 'entropy']\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "reg = RandomForestClassifier()\n",
    "# Define the hyperparameter configuration space\n",
    "space  = [Integer(10, 100, name='n_estimators'),\n",
    "            Integer(5, 50, name='max_depth'),\n",
    "          Integer(1, 64, name='max_features'),\n",
    "          Integer(2, 11, name='min_samples_split'),\n",
    "          Integer(1, 11, name='min_samples_leaf'),\n",
    "         Categorical(['gini', 'entropy'], name='criterion'),]\n",
    "# Define the objective function\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"accuracy\"))\n",
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=20, random_state=0)\n",
    "print(\"Accuracy:%.4f\" % -res_gp.fun)\n",
    "print(res_gp.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.9744\n",
      "[16.876434059259253, 'rbf']\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "reg = SVC(gamma='scale')\n",
    "space  = [Real(0.01, 50, name='C'),\n",
    "          Categorical(['linear','poly','rbf','sigmoid'], name='kernel'),\n",
    "         ]\n",
    "\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"accuracy\"))\n",
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=20, random_state=0)\n",
    "print(\"Accuracy:%.4f\" % -res_gp.fun)\n",
    "print(res_gp.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.9622\n",
      "[2]\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "reg = KNeighborsClassifier()\n",
    "space  = [Integer(1, 20, name='n_neighbors')]\n",
    "\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"accuracy\"))\n",
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=10, random_state=0)\n",
    "print(\"Accuracy:%.4f\" % -res_gp.fun)\n",
    "print(res_gp.x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 5: BO-TPE\n",
    "Bayesian Optimization with Tree-structured Parzen Estimator (TPE)\n",
    "\n",
    "**Advantages:**\n",
    "* Efficient with all types of HPs.\n",
    "* Keep conditional dependencies.\n",
    "\n",
    "**Disadvantages:**  \n",
    "* Poor capacity for parallelization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.76it/s, best loss: -0.9348679045482652]\n",
      "Random Forest: Hyperopt estimated optimum {'n_estimators': 95.0, 'max_features': 13.0, 'max_depth': 39.0, 'min_samples_split': 3.0, 'min_samples_leaf': 2.0, 'criterion': 0}\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'max_features': int(params['max_features']),\n",
    "        \"min_samples_split\":int(params['min_samples_split']),\n",
    "        \"min_samples_leaf\":int(params['min_samples_leaf']),\n",
    "        \"criterion\":str(params['criterion'])\n",
    "    }\n",
    "    clf = RandomForestClassifier( **params)\n",
    "    score = cross_val_score(clf, X, y, scoring='accuracy', cv=StratifiedKFold(n_splits=3)).mean()\n",
    "    #print(\"ROC-AUC {:.3f} params {}\".format(score, params))\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 100, 1),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"max_features\":hp.quniform('max_features', 1, 64, 1),\n",
    "    \"min_samples_split\":hp.quniform('min_samples_split',2,11,1),\n",
    "    \"min_samples_leaf\":hp.quniform('min_samples_leaf',1,11,1),\n",
    "    \"criterion\":hp.choice('criterion',['gini','entropy'])\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"Random Forest: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████| 20/20 [00:02<00:00,  7.18it/s, best loss: -0.9749661645191837]\n",
      "SVM: Hyperopt estimated optimum {'kernel': 2, 'C': 5.89740125613865}\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'C': abs(float(params['C'])), \n",
    "        \"kernel\":str(params['kernel'])\n",
    "    }\n",
    "    clf = SVC(gamma='scale', **params)\n",
    "    score = cross_val_score(clf, X, y, scoring='accuracy', cv=StratifiedKFold(n_splits=3)).mean()\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    'C': hp.normal('C', 0, 50),\n",
    "    \"kernel\":hp.choice('kernel',['linear','poly','rbf','sigmoid'])\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"SVM: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████| 10/10 [00:02<00:00,  4.34it/s, best loss: -0.968293886616605]\n",
      "KNN: Hyperopt estimated optimum {'n_neighbors': 3.0}\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_neighbors': abs(int(params['n_neighbors']))\n",
    "    }\n",
    "    clf = KNeighborsClassifier( **params)\n",
    "    score = cross_val_score(clf, X, y, scoring='accuracy', cv=StratifiedKFold(n_splits=3)).mean()\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    'n_neighbors': hp.quniform('n_neighbors', 1, 20, 1),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=10)\n",
    "print(\"KNN: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████| 10/10 [06:29<00:00, 38.92s/it, best loss: -1.0]\n",
      "ANN: Hyperopt estimated optimum {'activation': 1, 'epochs': 30.0, 'optimizer': 0, 'patience': 9.0, 'batch_size': 48.0, 'neurons': 60.0}\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        \"optimizer\":str(params['optimizer']),\n",
    "        \"activation\":str(params['activation']),\n",
    "        'batch_size': abs(int(params['batch_size'])),\n",
    "        'neurons': abs(int(params['neurons'])),\n",
    "        'epochs': abs(int(params['epochs'])),\n",
    "        'patience': abs(int(params['patience']))\n",
    "    }\n",
    "    clf = KerasClassifier(build_fn=ANN,**params, verbose=0)\n",
    "    score = -np.mean(cross_val_score(clf, X, y, cv=3, \n",
    "                                    scoring=\"accuracy\"))\n",
    "\n",
    "    return {'loss':score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    \"optimizer\":hp.choice('optimizer',['adam','rmsprop','sgd']),\n",
    "    \"activation\":hp.choice('activation',['relu','tanh']),\n",
    "    'batch_size': hp.quniform('batch_size', 16, 64, 16),\n",
    "    'neurons': hp.quniform('neurons', 10, 100, 10),\n",
    "    'epochs': hp.quniform('epochs', 20, 50, 10),\n",
    "    'patience': hp.quniform('patience', 3, 20, 3),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=10)\n",
    "print(\"ANN: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 6: PSO\n",
    "Partical swarm optimization (PSO): Each particle in a swarm communicates with other particles to detect and update the current global optimum in each iteration until the final optimum is detected.\n",
    "\n",
    "**Advantages:**\n",
    "* Efficient with all types of HPs.\n",
    "* Enable parallelization. \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Require proper initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 72.7099609375, 'max_features': 7.49072265625, 'max_depth': 28.31298828125, 'min_samples_split': 10.63525390625, 'min_samples_leaf': 5.6337890625, 'criterion': 0.22998046875}\n",
      "Accuracy:0.9255791726758763\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "data=X\n",
    "labels=y.tolist()\n",
    "# Define the hyperparameter configuration space\n",
    "search = {\n",
    "    'n_estimators': [10, 100],\n",
    "    'max_features': [1, 64],\n",
    "    'max_depth': [5,50],\n",
    "    \"min_samples_split\":[2,11],\n",
    "    \"min_samples_leaf\":[1,11],\n",
    "    \"criterion\":[0,1]\n",
    "         }\n",
    "# Define the objective function\n",
    "@optunity.cross_validated(x=data, y=labels, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,n_estimators=None, max_features=None,max_depth=None,min_samples_split=None,min_samples_leaf=None,criterion=None):\n",
    "    # fit the model\n",
    "    if criterion<0.5:\n",
    "        cri='gini'\n",
    "    else:\n",
    "        cri='entropy'\n",
    "    model = RandomForestClassifier(n_estimators=int(n_estimators),\n",
    "                                   max_features=int(max_features),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   min_samples_split=int(min_samples_split),\n",
    "                                   min_samples_leaf=int(min_samples_leaf),\n",
    "                                   criterion=cri,\n",
    "                                  )\n",
    "    #predictions = model.predict(x_test)\n",
    "    scores=np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"accuracy\"))\n",
    "    #return optunity.metrics.roc_auc(y_test, predictions, positive=True)\n",
    "    return scores#optunity.metrics.accuracy(y_test, predictions)\n",
    "\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 1.55078125, 'C': 47.998046875}\n",
      "Accuracy:0.9604833211865952\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "data=X\n",
    "labels=y.tolist()\n",
    "\n",
    "search = {\n",
    "    'C': (0,50),\n",
    "    'kernel':[0,4]\n",
    "         }\n",
    "@optunity.cross_validated(x=data, y=labels, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,C=None,kernel=None):\n",
    "    # fit the model\n",
    "    if kernel<1:\n",
    "        ke='linear'\n",
    "    elif kernel<2:\n",
    "        ke='poly'\n",
    "    elif kernel<3:\n",
    "        ke='rbf'\n",
    "    else:\n",
    "        ke='sigmoid'\n",
    "    model = SVC(C=float(C),\n",
    "                kernel=ke\n",
    "                                  )\n",
    "    #predictions = model.predict(x_test)\n",
    "    scores=np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"accuracy\"))\n",
    "    #return optunity.metrics.roc_auc(y_test, predictions, positive=True)\n",
    "    return scores#optunity.metrics.accuracy(y_test, predictions)\n",
    "\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 3.12451171875}\n",
      "Accuracy:0.968293886616605\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "data=X\n",
    "labels=y.tolist()\n",
    "\n",
    "search = {\n",
    "    'n_neighbors': [1, 20],\n",
    "         }\n",
    "@optunity.cross_validated(x=data, y=labels, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,n_neighbors=None):\n",
    "    # fit the model\n",
    "    model = KNeighborsClassifier(n_neighbors=int(n_neighbors),\n",
    "                                  )\n",
    "    scores=np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"accuracy\"))\n",
    "    return scores\n",
    "\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=10,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'optimizer': 1.614455714955763, 'activation': 0.41885608906506233, 'batch_size': 1.1755859375, 'neurons': 79.7763671875, 'epochs': 21.244529758913554, 'patience': 3.140998456489326}\n",
      "Accuracy:0.9907252828788723\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "data=X\n",
    "labels=y.tolist()\n",
    "\n",
    "search = {\n",
    "    'optimizer':[0,3],\n",
    "    'activation':[0,2],\n",
    "    'batch_size': [0, 2],\n",
    "    'neurons': [10, 100],\n",
    "    'epochs': [20, 50],\n",
    "    'patience': [3, 20],\n",
    "         }\n",
    "@optunity.cross_validated(x=data, y=labels, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,optimizer=None,activation=None,batch_size=None,neurons=None,epochs=None,patience=None):\n",
    "    # fit the model\n",
    "    if optimizer<1:\n",
    "        op='adam'\n",
    "    elif optimizer<2:\n",
    "        op='sgd'\n",
    "    else:\n",
    "        op='rmsprop'\n",
    "    if activation<1:\n",
    "        ac='relu'\n",
    "    else:\n",
    "        ac='tanh'\n",
    "    if batch_size<1:\n",
    "        ba=16\n",
    "    else:\n",
    "        ba=32\n",
    "    model = ANN(optimizer=op,\n",
    "                activation=ac,\n",
    "                batch_size=ba,\n",
    "                neurons=int(neurons),\n",
    "                epochs=int(epochs),\n",
    "                patience=int(patience)\n",
    "                                  )\n",
    "    clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "    scores=np.mean(cross_val_score(clf, X, y, cv=3, \n",
    "                                    scoring=\"accuracy\"))\n",
    "\n",
    "    return scores\n",
    "\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 7: Genetic Algorithm\n",
    "Genetic algorithms detect well-performing hyper-parameter combinations in each generation, and pass them to the next generation until the best-performing combination is identified.\n",
    "\n",
    "**Advantages:**\n",
    "* Efficient with all types of HPs.\n",
    "* Not require good initialization.\n",
    " \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Poor capacity for parallelization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using DEAP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1, 1, 1, 1, 1, 1] and maxint [89, 62, 44, 8, 9, 1] detected\n",
      "--- Evolve in 45927000 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin     \tmax     \tstd      \n",
      "0  \t10    \t0.898664\t0.871452\t0.920423\t0.0133659\n",
      "1  \t8     \t0.90345 \t0.883139\t0.919866\t0.00932254\n",
      "2  \t6     \t0.911408\t0.902059\t0.919866\t0.00498231\n",
      "3  \t6     \t0.914969\t0.904285\t0.919866\t0.00508078\n",
      "4  \t6     \t0.918976\t0.913189\t0.919866\t0.0020401 \n",
      "5  \t6     \t0.919588\t0.917084\t0.919866\t0.000834725\n",
      "Best individual is: {'n_estimators': 38, 'max_features': 3, 'max_depth': 16, 'min_samples_split': 6, 'min_samples_leaf': 3, 'criterion': 'gini'}\n",
      "with fitness: 0.9204229271007234\n",
      "{'n_estimators': 38, 'max_features': 3, 'max_depth': 16, 'min_samples_split': 6, 'min_samples_leaf': 3, 'criterion': 'gini'}\n",
      "Accuracy:0.9204229271007234\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': np.logspace(1,1.8,num = 10 ,base=20,dtype='int'),\n",
    "    'max_depth': np.logspace(1,2,num = 10 ,base=10,dtype='int'),\n",
    "    \"max_features\":np.logspace(0.2,1,num = 5 ,base=8,dtype='int'),\n",
    "    \"min_samples_split\":np.logspace(0.4, 1, num=5, base=10, dtype='int'), #[2, 3, 5, 7, 10],\n",
    "    \"min_samples_leaf\":np.logspace(0.1,1,num = 5 ,base=11,dtype='int'),\n",
    "    \"criterion\":['gini','entropy']\n",
    "}\n",
    "rf_params = {\n",
    "    'n_estimators': range(10,100),\n",
    "    \"max_features\":range(1,64),\n",
    "    'max_depth': range(5,50),\n",
    "    \"min_samples_split\":range(2,11),\n",
    "    \"min_samples_leaf\":range(1,11),\n",
    "    #Categorical(name='criterion', categories=['gini','entropy'])#\n",
    "    \"criterion\":['gini','entropy']\n",
    "}\n",
    "clf = RandomForestClassifier(random_state=0)\n",
    "# Set the hyperparameters of GA \n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"accuracy\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"Accuracy:\"+ str(ga1.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1, 2] and maxint [3, 999] detected\n",
      "--- Evolve in 4000 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin     \tmax     \tstd     \n",
      "0  \t10    \t0.906177\t0.760156\t0.974402\t0.089037\n",
      "1  \t7     \t0.949527\t0.753478\t0.974402\t0.0655796\n",
      "2  \t7     \t0.974402\t0.974402\t0.974402\t0        \n",
      "3  \t6     \t0.952977\t0.760156\t0.974402\t0.0642738\n",
      "4  \t4     \t0.974402\t0.974402\t0.974402\t0        \n",
      "5  \t3     \t0.971341\t0.943795\t0.974402\t0.00918197\n",
      "Best individual is: {'kernel': 'rbf', 'C': 26.705081969400556}\n",
      "with fitness: 0.9744017807456873\n",
      "{'kernel': 'rbf', 'C': 26.705081969400556}\n",
      "Accuracy:0.9744017807456873\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "rf_params = {\n",
    "    'C': np.random.uniform(0,50,1000),\n",
    "    \"kernel\":['linear','poly','rbf','sigmoid']\n",
    "}\n",
    "clf = SVC(gamma='scale')\n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"accuracy\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"Accuracy:\"+ str(ga1.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1] and maxint [18] detected\n",
      "--- Evolve in 19 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin     \tmax    \tstd       \n",
      "0  \t10    \t0.955092\t0.946578\t0.96828\t0.00705242\n",
      "1  \t7     \t0.962994\t0.951029\t0.96828\t0.00558289\n",
      "2  \t5     \t0.967501\t0.964385\t0.96828\t0.00155815\n",
      "3  \t8     \t0.967891\t0.964385\t0.96828\t0.00116861\n",
      "4  \t8     \t0.966221\t0.947691\t0.96828\t0.00617696\n",
      "5  \t6     \t0.96828 \t0.96828 \t0.96828\t0         \n",
      "Best individual is: {'n_neighbors': 3}\n",
      "with fitness: 0.9682804674457429\n",
      "{'n_neighbors': 3}\n",
      "Accuracy:0.9682804674457429\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "rf_params = {\n",
    "    'n_neighbors': range(1,20),\n",
    "}\n",
    "clf = KNeighborsClassifier()\n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"accuracy\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"Accuracy:\"+ str(ga1.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1, 1, 1, 1, 1, 1] and maxint [1, 1, 2, 16, 2, 89] detected\n",
      "--- Evolve in 55080 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin    \tmax     \tstd      \n",
      "0  \t10    \t0.985031\t0.93044\t0.999444\t0.0217448\n",
      "1  \t6     \t0.998164\t0.995548\t0.999444\t0.00174296\n",
      "2  \t4     \t0.999444\t0.999444\t0.999444\t1.11022e-16\n",
      "3  \t3     \t0.999444\t0.999444\t0.999444\t1.11022e-16\n",
      "4  \t6     \t0.999444\t0.999444\t0.999444\t1.11022e-16\n",
      "5  \t8     \t0.999444\t0.999444\t0.999444\t1.11022e-16\n",
      "Best individual is: {'activation': 'relu', 'epochs': 50, 'optimizer': 'sgd', 'patience': 14, 'batch_size': 16, 'neurons': 64}\n",
      "with fitness: 0.9994435169727324\n",
      "{'activation': 'relu', 'epochs': 50, 'optimizer': 'sgd', 'patience': 14, 'batch_size': 16, 'neurons': 64}\n",
      "Accuracy:0.9994435169727324\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop','sgd'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':range(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':range(3,20)\n",
    "}\n",
    "clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "# Set the hyperparameters of GA    \n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"accuracy\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"Accuracy:\"+ str(ga1.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using TPOT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n",
      "Generation 1 - Current Pareto front scores:\n",
      "-1\t0.9248510704682911\tRandomForestClassifier(CombineDFs(input_matrix, input_matrix), RandomForestClassifier__criterion=gini, RandomForestClassifier__max_depth=42, RandomForestClassifier__max_features=24, RandomForestClassifier__min_samples_leaf=2, RandomForestClassifier__min_samples_split=2, RandomForestClassifier__n_estimators=110)\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "-1\t0.9331983437951373\tRandomForestClassifier(input_matrix, RandomForestClassifier__criterion=gini, RandomForestClassifier__max_depth=99, RandomForestClassifier__max_features=22, RandomForestClassifier__min_samples_leaf=1, RandomForestClassifier__min_samples_split=5, RandomForestClassifier__n_estimators=192)\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "-1\t0.9331983437951373\tRandomForestClassifier(input_matrix, RandomForestClassifier__criterion=gini, RandomForestClassifier__max_depth=99, RandomForestClassifier__max_features=22, RandomForestClassifier__min_samples_leaf=1, RandomForestClassifier__min_samples_split=5, RandomForestClassifier__n_estimators=192)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTClassifier(config_dict={'sklearn.ensemble.RandomForestClassifier': {'n_estimators': range(20, 200), 'max_features': range(1, 64), 'max_depth': range(10, 100), 'min_samples_split': range(2, 11), 'min_samples_leaf': range(1, 11), 'criterion': ['gini', 'entropy']}},\n",
       "        crossover_rate=0.1, cv=3, disable_update_check=False, early_stop=5,\n",
       "        generations=3, max_eval_time_mins=5, max_time_mins=None,\n",
       "        memory=None, mutation_rate=0.9, n_jobs=1, offspring_size=5,\n",
       "        periodic_checkpoint_folder=None, population_size=10,\n",
       "        random_state=None, scoring='accuracy', subsample=1.0,\n",
       "        template=None, use_dask=False, verbosity=3, warm_start=False)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Random Forest\n",
    "from tpot import TPOTClassifier\n",
    "# Define the hyperparameter configuration space\n",
    "parameters = {\n",
    "        'n_estimators': range(20,200),\n",
    "    \"max_features\":range(1,64),\n",
    "    'max_depth': range(10,100),\n",
    "    \"min_samples_split\":range(2,11),\n",
    "    \"min_samples_leaf\":range(1,11),\n",
    "    \"criterion\":['gini','entropy']\n",
    "             }\n",
    "# Set the hyperparameters of GA                 \n",
    "ga2 = TPOTClassifier(generations= 3, population_size= 10, offspring_size= 5,\n",
    "                                 verbosity= 3, early_stop= 5,\n",
    "                                 config_dict=\n",
    "                                 {'sklearn.ensemble.RandomForestClassifier': parameters}, \n",
    "                                 cv = 3, scoring = 'accuracy')\n",
    "ga2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n",
      "Generation 1 - Current Pareto front scores:\n",
      "-1\t0.9604833211865952\tSVC(input_matrix, SVC__C=44.89175174937672, SVC__kernel=poly)\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "-1\t0.9604833211865952\tSVC(input_matrix, SVC__C=44.89175174937672, SVC__kernel=poly)\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "-1\t0.9604833211865952\tSVC(input_matrix, SVC__C=44.89175174937672, SVC__kernel=poly)\n",
      "\n",
      "\n",
      "The optimized pipeline was not improved after evaluating 5 more generations. Will end the optimization process.\n",
      "\n",
      "TPOT closed prematurely. Will use the current best pipeline.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTClassifier(config_dict={'sklearn.svm.SVC': {'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'C': array([29.68993, 40.91101, ..., 20.12904, 32.1227 ])}},\n",
       "        crossover_rate=0.1, cv=3, disable_update_check=False, early_stop=5,\n",
       "        generations=3, max_eval_time_mins=5, max_time_mins=None,\n",
       "        memory=None, mutation_rate=0.9, n_jobs=1, offspring_size=5,\n",
       "        periodic_checkpoint_folder=None, population_size=10,\n",
       "        random_state=None, scoring='accuracy', subsample=1.0,\n",
       "        template=None, use_dask=False, verbosity=3, warm_start=False)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#SVM\n",
    "from tpot import TPOTClassifier\n",
    "\n",
    "parameters = {\n",
    "    'C': np.random.uniform(0,50,1000),\n",
    "    \"kernel\":['linear','poly','rbf','sigmoid']\n",
    "             }\n",
    "               \n",
    "ga2 = TPOTClassifier(generations= 3, population_size= 10, offspring_size= 5,\n",
    "                                 verbosity= 3, early_stop= 5,\n",
    "                                 config_dict=\n",
    "                                 {'sklearn.svm.SVC': parameters}, \n",
    "                                 cv = 3, scoring = 'accuracy')\n",
    "ga2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n",
      "Generation 1 - Current Pareto front scores:\n",
      "-1\t0.9644039680147021\tKNeighborsClassifier(CombineDFs(input_matrix, input_matrix), KNeighborsClassifier__n_neighbors=4)\n",
      "-2\t0.968853170732936\tKNeighborsClassifier(KNeighborsClassifier(input_matrix, KNeighborsClassifier__n_neighbors=15), KNeighborsClassifier__n_neighbors=3)\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "-1\t0.968293886616605\tKNeighborsClassifier(input_matrix, KNeighborsClassifier__n_neighbors=3)\n",
      "-2\t0.968853170732936\tKNeighborsClassifier(KNeighborsClassifier(input_matrix, KNeighborsClassifier__n_neighbors=15), KNeighborsClassifier__n_neighbors=3)\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "-1\t0.968293886616605\tKNeighborsClassifier(input_matrix, KNeighborsClassifier__n_neighbors=3)\n",
      "-2\t0.968853170732936\tKNeighborsClassifier(KNeighborsClassifier(input_matrix, KNeighborsClassifier__n_neighbors=15), KNeighborsClassifier__n_neighbors=3)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTClassifier(config_dict={'sklearn.neighbors.KNeighborsClassifier': {'n_neighbors': range(1, 20)}},\n",
       "        crossover_rate=0.1, cv=3, disable_update_check=False, early_stop=5,\n",
       "        generations=3, max_eval_time_mins=5, max_time_mins=None,\n",
       "        memory=None, mutation_rate=0.9, n_jobs=1, offspring_size=5,\n",
       "        periodic_checkpoint_folder=None, population_size=10,\n",
       "        random_state=None, scoring='accuracy', subsample=1.0,\n",
       "        template=None, use_dask=False, verbosity=3, warm_start=False)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#KNN\n",
    "from tpot import TPOTClassifier\n",
    "\n",
    "parameters = {\n",
    "    'n_neighbors': range(1,20),\n",
    "             }\n",
    "               \n",
    "ga2 = TPOTClassifier(generations= 3, population_size= 10, offspring_size= 5,\n",
    "                                 verbosity= 3, early_stop= 5,\n",
    "                                 config_dict=\n",
    "                                 {'sklearn.neighbors.KNeighborsClassifier': parameters}, \n",
    "                                 cv = 3, scoring = 'accuracy')\n",
    "ga2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "tf36",
   "language": "python",
   "name": "tf36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
